posted 08-24-2010 04:16 PM            

Bootstrap training of the component and criterion norms improves overall decision accuracy and produces a model that will generalize more effectively (overfit less) than the model that was trained without bootstrapping.

The non-bootstrapped model, while it does have zero inconclusives for deceptive cases, is an situation of overfitting, It may not generalize as effectively to new data.

We'll see.

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 08-24-2010 04:19 PM            

The point is that bootstrapping does assume the data are representative, but does NOT cherry-pick cases or overtrain/overfit a model. In fact it is , bootstrapping is used to produce models that generalize more effectively to new data and overfits less.

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 08-24-2010 04:39 PM            

Once again the bootstrapped model shows an incremental improvement over the naive/non-bootstrapped model.

Now, you might have no false negative errors... but then again, is it realistic and wise to make that kind of assumption and sell it to our customers and communities. They're gonna feel awfully mad and feel like we deceived them if there is ever a day when someone does observe one of them FNs.

Previous research does tell us that FNs are rare, and that most errors are PFs. Krapohl (2005) reported data that included 2.7% FNs. Blackwell (1999) had 3.3% FNs. Using these figures we can use the laws of small numbers and Poisson analysis to determine whether our observance of zero FNs in a sample of 35 deceptive cases is statistically significant - it is not

The Poisson probability of observing zero FN errors in a sample of 35 deceptive cases is calculated about 37%, which is not statistically significant.

So, a sample of 35 does lack statistical power to draw strong conclusion. If however (that is, IF) we were to observe zero FN errors in a sample of say N=240 or more deceptive cases, the Poisson probability would be estimated at less that 0.01% - which would be statistically significant. So it seems that a sample of approximately N=300 might provide us with statistical power to make generalizable conclusions.

At this point the bootstrapped trained OSS-3 model seems to be working better than a naïve model.

I dunno folks, perhaps you can not like bootstrapping and monte carlo models for some odd reason of you own, but it seems to work OK. They are also rather simple considering the complexity of the problems they can effectively address. Their effectiveness is probably why they are used so often.

Would seem rather negligent and arrogant of us not to learn to use the same kinds of good computerized tools and statistical models for studying data as are used on other fields of science.

Would also seem rather arrogant of us to use computer algorithms and statistical models with no accountability or knowledge about wtf they do.

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 08-24-2010 09:20 PM         

As always, I enjoy reading your posts. I am definately in the "WTF" crowd when it comes to "Bootstraping" and "Barf-O-Roni". Is there a way to publish this info in a way that Sacket and I can understand it? I see this same type of scientific jargon creeping into ASTM standards and I am afraid that many of our "Practioners" are going to be kept after school in an attempt to get caught up. I agree that we all need more education but I think many of us are having trouble keeping up.

You produce great stuff-I just wish I was able to follow more of it!

THX

posted 08-24-2010 10:18 PM            

It is most probably NOT you or Sackett that will have trouble keeping up with this. Every time I want to argue with Sackett I find myself agreeing with him more than not.

Mostly, I just want you to know about it. You don't have to know the procedural details, any more than I really need to know how my automatic transmission works.

But you should know that it is legit, and anyone who does feel geeky and so inclined should be able to get more information.

We should all be able to be confident that there is more than just one or two people in the world who are capable or permitted to know the WTF details - with our auto-transmissions and our polygraph decision theory.

Anyway, the ASTM stuff is not just ASTM...

We are going to be facing increasing expectations to account for our work in scientific terms.

What will not be acceptable and will not win the long game of survival and progress is for our expertise to be limited to the checklist kind of expertise - in which we are expert at following a checklist and not much else.

There is no great need to worry much about not actually having answers. What we need to do is just keep learning a few new words and concepts, and know where to look in the recently published polygraph studies for the simple answers to the seemingly difficult scientific questions.

We have been working persistently to develop and publish answers to the seemingly difficult questions - you just have to look them up, and there really aren't that many of them.

The real worry will be the impulse to engage in make-believe science in which we do not use statistical decision theory, and instead engage in all kinds of amateur psychologizing and amateur linguistics to attempt to anchor the construct validity of the polygraph test. Psychological and linguistic ideas will eventually require proof, or will have to be discarded - because that is how the science game is played.

Another impulse will be to play a game of follow the leader instead of follow the data. The people most a risk for this are those who truly cannot think for themselves and lack the ability to make sense out of information - they will need to be told what to do, else they will feel paralyzed by their own lack of understanding.

Probably the next biggest concern is whether or not we continue to teach and emphasize arcane ideas that are exceedingly complex and already dis-proven - just because we were taught that stuff so many years ago. To do this will be to foster a kind of professional in-breeding that could easily stagnate knowledge and progress in a whole profession for say 30 or 40 years.

The good new for all of us, is that a lot of smart people have been doing important work for decades already: Backster, Kircher/Raskin/Honts and others at Utah, Barland, Senter, Krapohl and anyone who I apologize for forgetting. It is an interesting time and opportunity to put it all together and fill in a few remaining pieces. If we do this right we will find ourselves with many more answers than we thought possible, and the increased acceptance of polygraph will be inevitable - because it will be evident that we work on and talk about the same conceptual problems as our related fields of science.

.02

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 08-25-2010 07:02 AM         

Well said. Thank you!

posted 08-25-2010 07:30 PM            

Congrats on winning a seat on the APA Board. I look forward to a productive year.

posted 08-26-2010 06:10 AM            

Mark Handler here thanking you for stepping up to the plate and shining a light on a problem that some would let smolder or fester. You are really one of the brave guys out there who are not afraid to say "I don't know, but I want to learn if I can." I promise you I can help you understand as much of this as you need to know. I am light years from reaching Raymond's level and I lean on him heavily to help understand this stuff. I feel it is only the right thing to do and give back some of what he has taught me. Yesterday I agreed to have a state organization come to my home for training (pizza and ESS, it doesn't get any better than that).

I am the one who tried to infuse these concepts into the ASTM standards and given the NAS recommendations, the timing was uncanny. Unfortunately the work item has received a number of negative votes accompanied mostly with language to the effect that "we don't need to know this stuff", "I don't understand your point", "I shouldn't have to make an assessment of a subject's fitness for testing, I am not a psychologist", or my favorite "This doesn't belong here". I am surprised and not surprised at the reactions. Some are legitimate fear-based, knee-jerks which can be fixed if we just help people work their way through these concepts.

Ray and I will write a paper on decision theory that will break it down (OK Don, you got us-I just hope Ray doesn't mind me speaking for him but I know him well and know his heart). Unfortunately some of the people voting against these things seem to resist for unknown reasons, and those people may be tougher. Educating a person is one thing, changing an strongly held attitude is another. I will say that I am not going to attempt to revise the ASTM standard again. I give up and someone else will have to try. I will not keep banging my head against the wall on it and let someone else take this torch.

Ray and Don, thanks for continuing to inspire us to turn the Titanic. I remember from my submarine days the old beat up tug boats that steered my boat into the harbor. Being a tugboat isn't a glamorous job, but it has a purpose.

posted 08-26-2010 10:25 AM            

You've got to take a look at this:
http://www.ascld.org/files/releases/NAS%20Executive%20Summary%20090218.pdf

Here's a quote from the above document:

quote:

---

Insufficient Education and Training
Forensic science examiners need to understand the principles, practices, and contexts of scientific methodology, as well as the distinctive features of their specialty. Ideally, training should
move beyond apprentice-like transmittal of practices to education based on scientifically valid principles. In addition to the practical experience and learning acquired during an internship, a
trainee should acquire rigorous interdisciplinary education and training in the scientific areas that constitute the basis for the particular forensic discipline and instruction on how to document and report the analysis. A trainee also should have working knowledge of basic quantitative calculations, including statistics and probability, as needed for the applicable discipline.

To correct some of the existing deficiencies, it is crucially important to improve undergraduate and graduate forensic science programs. Legitimization of practices in forensic disciplines must be based on established scientific knowledge, principles, and practices, which are best learned through formal education. Apprenticeship has a secondary role, and under no circumstances can it supplant the need for the scientific basis of education in and the practice of forensic science.

---

What you find from reading where forensic science is going in the future, is that our current method of operation is a problem. If we don't fall in line with what's coming, we're going to be relegated to irrelevant or "junk" science. If that happens, polygraph is dead. How do you defend against a lawsuit in which polygraph is involved when it doesn't make the "list"? You can't, and we won't.

posted 08-26-2010 04:30 PM         

If your post is directed to me, you will need to write slower! (LOL)

I agree with you, Mark and Ray. As a forensic science, we either move forward or expire. I am just the kind of guy who not only needs to know what we have, but how we got it. I also need to know how and why it works. I am hoping that Mark's paper will help answer these questions when it comes to moving the profession forward. There are very few of us in the WTF crowd that can keep up with the brilliance of Mark, Don, Ray, Cleve and yourself. We all want to move the science forward but need to do so in a manner that we can comprehend and absorb.

Keep up the good work guys !

Ted

posted 08-26-2010 07:23 PM            

Mark here again (in between kid duty).

Brilliant, I am not though if asked about being dedicated to moving the profession forward, I must confess my guilt.

Without trying to make this the mutual appreciation blog, I really do want to thank you for having the fortitude to stand up and ask for help.

When Ray and I started to develop the ESS, we went about it as we did with OSS. We broke it down to the smallest parts and worked upward. We have been working with Stu Senter and Don Krapohl on this since 2007 and we are finally getting to the point where we understand it and are ready to publish what we have learned. Guys like Krapohl, Honts, Senter and Kircher have known it all along but have been building their case slowly.

I agree with Barry and Don K. completely and know that the days of the "polygraph practitioner" are over. The proverbial train has left the station and it is time to get on board or get left behind. I don't say this in a mean way, hell I open my home to anyone interested in sharing these ideas with me. In the "old days" examiners could sit back and say "I did it this way because DACA says so" and that was a good enough wall to hide behind. Now that we have data we are no longer able to take that posture.

We have to be able to describe in simple terms what we do, how we do it, why we think it works, how sure we are of our answer, what are the chances we made a mistake and what kind of research supports what we do. There is incredible value in statistical confidence in our result. It is the difference between the utility of the test and the utility of the test result. They really do differ. The test result should have utility to the end-user and gate keepers. It should add incremental validity to the purpose for which the test was administered.

Having this new-found knowledge is a double-edged sword. We can't fall back on the sins of our fore-fathers anymore and just play "follow the leader". We have to answer questions in the light of the data because ultimately the data will carry the day.

I was a cop for many years and I feel I can teach these concepts to any police or private examiner. I teach them to basic polygraph students when I am blessed enough to have the privileged. As I said in my earlier post, I am far from the expert but I do have a working knowledge and am willing to share it with others.

I will check with Don but perhaps the APA magazine is a better venue for a more "friendly" written description of some of these concepts. We are actively recruiting examiners to go out and teach the ESS at seminars. Ray is tied up with his day job (Lafayette) and I am tied up with mine (Maddie, Travis and Jacob).

posted 08-27-2010 07:26 AM            

Check your email for the latest draft of the ESS paper we are working on and the PPT we put together for Rick Kurtz.

Make a list of questions as you go through.
Examples may be;

What do they mean by sensitivity, specificity, PPV, NPV, truthful inconclusives, deceptive inconclusives, false negative results, false positive results, p-values, alpha, criteria, transformation, decision rules, statistical cut scores, etc.

posted 09-19-2010 04:46 PM         

I am still chewing on both of the papers you sent. GREAT STUFF but still a little over my head. I don't think I am alone. I am not big on statistics or I would have taken a Stat class and completed my Masters Degree!

I would love to see a seminar presentation by you guys titled "ESS For Dummies" at the next seminar. I think it would be a standing room only hit and would be well received. I might have to sit through it twice.

If we can all grasp the concepts that the "Big Dogs" at putting out there, then we can all move this science forward together!

Keep up the good work.

posted 09-19-2010 05:47 PM         

T...E...D

posted 09-20-2010 07:59 AM            

Thanks Ray! You took a really boring but absolutely necessary subject, statistics and made it interesting and thought provoking and that's no easy task. ESS rocks!!!!

I have come to the conclusion, after listening to Ray that I should never write a polygraph report in which I state that "Johnny Badguy was deceptive when answering the relevant questions."

posted 09-20-2010 04:07 PM         

I just bought "Statistics for Dummies" for 27 clams. Not a bad deal. Below is what the book includes. I hope Ray, Mark and the rest of the gang are prepared for what happens when the WTF crowd dun git edjicated a litle bit!

Ted

In the numbers explosion all around us in our modern-day dealings, the buzzword is data, as in, “Do you have any data to support your claim?” “The data supported the original hypothesis that . . .” and “The data bear this out. . . .” But the field of statistics is not just about data. Statistics is the entire process involved in gathering evidence to answer questions about the world, in cases where that evidence happens to be numerical data.
Statistics For Dummies is for everyone who wants to sort through and evaluate the incredible amount of statistical information that comes to them on a daily basis. (You know the stuff: charts, graphs, tables, as well as headlines that talk about the results of the latest poll, survey, experiment, or other scientific study.) This book arms you with the ability to decipher and make important decisions about statistical results, being ever aware of the ways in which people can mislead you with statistics. Get the inside scoop on number-crunching nuances, plus insight into how you can

•Determine the odds
•Calculate a standard score
•Find the margin of error
•Recognize the impact of polls
•Establish criteria for a good survey
•Make informed decisions about experiments
This down-to-earth reference is chock-full of real examples from real sources that are relevant to your everyday life: from the latest medical breakthroughs, crime studies, and population trends to surveys on Internet dating, cell phone use, and the worst cars of the millennium. Statistics For Dummies departs from traditional statistics texts, references, supplement books, and study guides in the following ways:

•Practical and intuitive explanations of statistical concepts, ideas, techniques, formulas, and calculations.
•Clear and concise step-by-step procedures that intuitively explain how to work through statistics problems.
•Upfront and honest answers to your questions like, “What does this really mean?” and “When and how I will ever use this?”
Chances are, Statistics For Dummies will be your No. 1 resource for discovering how numerical data figures into your corner of the universe.

posted 09-21-2010 10:15 AM            

The "for dummies" books are well beyond "dummy" level, but they are written in everyday language and make concepts rather easy to understand.

Here's part of the ad from Amazon on the second in the series:

quote:

---

Begin with the basics — review the highlights of Stats I and expand on simple linear regression, confidence intervals, and hypothesis tests

Start making predictions — master multiple, nonlinear, and logistic regression; check conditions; and interpret results

Analyze variance with ANOVA — break down the ANOVA table, one-way and two-way ANOVA, the F-test, and multiple comparisons

Connect with Chi-square tests — examine two-way tables and test categorical data for independence and goodness-of-fit

Leap ahead with nonparametrics — grasp techniques used when you can't assume your data has a normal distribution

---

These are the things you are reading about in many of the current writings, and that, I presume, is what you better want to understand. Remember: in polygraph we deal more with categorical and nonparametric data, and that's studied after grasping how to deal with parametric ("normal") data. You'll need to understand both (to some extent), so don't cheat yourself.

posted 09-21-2010 10:25 AM         

Sorry for the rant-rough night last night. I'll try to get through the first book and go from there. Thanks for your suppor!

posted 09-22-2010 09:41 AM         

You might be referring to: Statistics in a nutshell - authors: Sarah Boslaugh, Paul Andrew Watters

posted 09-23-2010 01:01 PM            

I have not worked through any of the equations, so I cannot say whether there are mistakes or not. However, many college textbooks and published peer reviewed papers I have read have had errors in them.

The positive reviews, which outnumber the negative, can be read here: http://www.ama zon.com/Statistics-Nutshell-Desktop-Reference-OReilly/product-reviews/0596510497/ref=cm_cr_dp_hist_5?ie=UTF8&showViewpoints=0&filterBy=addFiveStar

I have read several books on statistics. So far, I have found that this book, compared to the others I have read, is a good reference book. It supplies information for each area you are referencing and provides insight on area's application.

Here is the list of the confirmed errata, which is suppose to be fixed in subsequent print and most are reported as fixed:
http://oreilly.com/catalog/errata.csp?isbn=9780596510497

posted 09-23-2010 08:01 PM            

A couple others are the Orange books - Statistics for Idiots (or something like that), they are competitors to the Dummies books.

And Statistics Demystified.

However, they will be boring.

The best way to learn about this stuff is to apply it to a problem in which you are actually interested.

You could use the books as reference only (and don't try to read them like novels or textbooks), and start to ask questions about the concepts you see on the printed OSS-3 report, and the ESS NPC Guide.

The math used in statistics is mostly just basic algebra. Statistics is really about conceptual problems. So conversation, questions, and discussion (explantation) goes a long way.

If the problem is interesting it does not take long for the concepts, issues and solutions to start to make sense.

Chad Russel was asking questions last week and concluded that a Monte Carlo is a little like playing fantasy baseball. We study a problem by building a model with whatever statistical and measurement data we have. Then we expand our knowledge and develop the model further from there.

We are approaching a point at which the majority of issues are identified for now, and we could start to summarize them in a couple of pages of cliff's notes - describing the issues in the procedures and results we work with.

This is the stuff that differentiates real professionals from the hacks who get there polygraph training in a box and consider themselves to be experts because the know ho to follow the checklist.

Anyway, thanks for your interest everyone. What we know today is actually the product of decades of tedious thinking. But we are closer than we think to a solid position if we just follow the data, follow the science.

One of the biggest mistakes some professions and some professionals make is call "slothful induction," which is when we have the data and have the evidence and ignore it in favor of some other belief.

r

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 09-23-2010 08:27 PM            

Let me agree with Ray and emphasize a point. The books are good. I like the for dummies - or idiots, etc - because you can sit down and read them and be educated and entertained at the same time. You won't get that in a college introductory text. However, as Ray said, it begins to make more sense when you have to grab a book to work your way through a problem.

posted 09-23-2010 08:46 PM            

Regardless, it is still a good read and covers a lot of what we do.

posted 10-09-2010 12:32 PM         

I am really hoping that Ray, Barry or Mark will be offering a class on this stuff at AAPP. I have heard nothing but great comments about the class Ray put on a APA.

posted 10-10-2010 05:47 AM            

I will check with Jimmy and others, but I think this is the agenda.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 10-19-2010 02:54 PM         

Talk slow and save me a front row seat!

posted 10-22-2010 08:57 AM            

I think that is find as a general and tentative statement.

In science we assume we know nothing, and we assume our own hypothesis (professional opinion) to be wrong until we can prove there is no other explanation.

Here is an accuracy profile from 7604 scored results of 1094 confirmed exams, from 18 cohorts of 119 examiners (experienced and inexperienced combined) who scored 12 different samples.

You can see that the decision policy makes a big difference.

What we found, when evaluating 30 different sets of scores was the 50 to 70% (mean = 60%) of truthful people will have at least one non-positive subtotal score. So, the two-stage rules is probably responsible for a a lot of the difference you see here.

ESS does seem to produce slightly better kappa scores for interrater reliability, when comparing inexperienced scorers using ESS to experienced scorers using 7-position models. But the difference does not appear to be significant.

Here you can see a direct comparison of ESS with the 7-position and 3 position models, with a group of inexperienced scorers.

What we see is that inexperienced scorers using ESS tend to score about as well as experienced examiners using the more complex 7-position model. There is some evidence, but we are not yet sure yet, that experienced scorers using ESS would perform better than the inexperienced scorers. One study in Maryland had outstanding results, and we hope to see that published soon. Here is some other data.

You can see the performance of the experienced people is a little better than the combined groups of experienced and inexperienced.

As always, more data helps us answer more questions.

What we need to avoid is the temptation to answer our questions with our expert opinions without data.

As always,

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 10-27-2010 08:12 PM            

What is more, ESS has monte carlo norms and can provide a calculated statistical classifier for three question ZCT and You Phase exams.

More still, ESS is the only method that includes a statistical solution to a statistical error estimate, classifier and p-value for multi-issue screening exams, including mulit-issue techniques with 2, 3 or 4 RQs.

Using the normative data, you can select in advance your desired or required level of precision for decision accuracy, inconclusive rates, and/or tolerance for error.

In other words decide how accurate you want your test to be and ESS will tell you what cutscore to use.

Or, using your numerical score to calculate the level of precision or error for you test result.

In other words, ESS can answer some of the very demanding questions that scientists (like those at the NAS and AAFS) ask: what it the level of statistical significance of your manual score?

If you don't quite understand this question, I invite you to contact me and get information about the ESS.

Best, of all, ESS is not new. It is based on decades of work by the same smart scientific thinkers you have been hearing about for a long time. Better yet, it is easy to learn and use, requires no printing or mechanical measurements and empowers polygraph field examiners to approach the polygraph with an evidence based model.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-14-2011 06:51 AM            

2. Do I understand correctly that with the OSS-3 (using two-stage rules, grand mean - inconclusive) and with the ESS (-7 for spot, overall total - NDI) we can make the DI decision with significant reaction to only one question in the signle-issue test despite that in such test a subject must answer all of the questions truthfully or all deceptively?

3. Do I understand correctly that with the OSS-3 we can make the NDI decision with the multi-facet test using grand mean rules (No Significant Reactions) despite that, for example, a subject could be deceptive to one question while being truthful to others?

4. Are there differences for the OSS-3 between the single-issue test and multi-facet test with using grand mean or two-stage rules? When I change a type of the test from the single-issue to multi-facet there is no differences in results.

Thanks.